Racial violence, particularly against black people, is also on the rise.
The report, by the Federal Commission Against Racism and an anti-racist NGO, said both recent migrants and people with Swiss citizenship were affected, and the majority of victims are men. The racist attacks are mainly serious verbal abuse, it said.
Incidents often happen at work or in other public areas, but also in people's dealings with the police.
"Most of the perpetrators were from a higher socio-economic background," the report said. "It is striking how often those accused (or committing racist acts) were in socio-economic positions of power and used this position, directly or indirectly against the victims." the report said.
The commission, together with Humanrights.ch, warns that the unreported cases of abuse are likely to be many more than those included in the report.
The report calls on Switzerland's cantons to improve measures protecting people against discrimination, and blames politicians for the rise in anti-Muslim sentiment.
"In recent years a number of MPs have been employed in denying the presence of racism in our society," the report said.
"In doing so, they are creating a favourable environment for exclusion and xenophobia."
The groups said votes to ban the building of minarets and send back foreign criminals, initiated by the right-wing Swiss People's Party, were directly linked to the rise in cases of anti-Muslim harassment.
"After the vote on minarets (in November 2009) Muslims were increasingly targeted," commission director Doris Angst said.
End item/ 129